[Storage Index Adapter] Fix index template creation on Serverless by viduni94 · Pull Request #264760 · elastic/kibana

viduni94 · 2026-04-21T15:44:21Z

Summary

Fixes index template creation on Serverless for indices kibana-evaluation-datasets, kibana-evaluation-dataset-examples).

PR #263096 added auto_expand_replicas and number_of_shards to index templates in StorageIndexAdapter. Serverless ES rejects these settings on non-system indices with an illegal_argument_exception, while hidden indices (e.g.: used by Streams) are unaffected because Kibana manages them as system indices.

Dataset upsert error for Kibana evaluation runs

Error in logs:

Failed to upsert evaluation dataset: ResponseError: illegal_argument_exception
	Root causes:
		illegal_argument_exception: Settings [index.auto_expand_replicas,index.number_of_shards] are not available when running in serverless mode

Fix

The changes were introduced in three tiers to detect serverless environments for index template settings:

Explicit detection - Introduced a new isServerless option in StorageIndexAdapterOptions. When provided, the adapter skips or includes settings without any extra calls.
Proactive - if isServerless is not provided, the adapter calls esClient.info() on the first write and checks version.build_flavor. The result is cached for the adapter's lifetime.
Reactive - if both above are unavailable (e.g.: info() fails due to insufficient privileges), the adapter catches the illegal_argument_exception on the first write, retries without settings, and caches the result.

The Evals plugin passes isServerless explicitly because the evals route handler creates StorageIndexAdapter with esClient.asCurrentUser, which is scoped to the caller's API key. This API key may lack the monitor cluster privilege needed for esClient.info(), making tier 2 unreliable. There buildFlavor is passed from the plugin context.

Test Plan

Deploy the fix to a serverless project from this PR
Create a config file (e.g.: config.testcluster.json) and add the serverless project URL as the dataset target
Run evals with node scripts/evals start --suite significant-events --project eis-anthropic-claude-4-6-sonnet --judge eis-google-gemini-3-1-pro --export-profile local --datasets-profile testcluster

With this fix, the dataset upsert works as expected

Checklist

Unit or functional tests were updated or added to match the most common scenarios
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

github-actions · 2026-04-21T15:44:33Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

viduni94 · 2026-04-21T15:54:54Z

 # Disable the embedded Dev Console
 console.ui.embeddedEnabled: false
+
+xpack.evals.enabled: true


Added this to test the changes against the serverless test deployment from this PR as the evals golden cluster is broken.

I will be removing this line before merging this PR.

flash1293 · 2026-04-21T15:59:34Z

Thanks @viduni94 , this is a very good catch - I'm very surprised our existing tests didn't capture it though... what's going on here? Do you happen to know @rudolf ?

In terms of the fix - we should proactively check whether we are running in serverless mode and decide based on that instead of waiting for the error and reacting to it.

viduni94 · 2026-04-21T16:14:38Z

Thanks @viduni94 , this is a very good catch - I'm very surprised our existing tests didn't capture it though... what's going on here? Do you happen to know @rudolf ?

In terms of the fix - we should proactively check whether we are running in serverless mode and decide based on that instead of waiting for the error and reacting to it.

Thanks @flash1293
Updated the fix to be proactive.
Let me know what you think.

elastic-vault-github-plugin-prod · 2026-04-21T17:33:53Z

Run Metadata

Triggered by: Issue #250
Elasticsearch image tag: 9.5.0-SNAPSHOT
Kibana image: docker.elastic.co/kibana-ci/kibana-serverless:pr-264760-05c6a8c37647
Date: 2026-04-21
PR: elastic/kibana#264760 — [Storage Index Adapter] Fix index template creation on Serverless
Mode: PR-targeted (journeys generated from PR diff)
Journeys executed: 8
Passed: 8
Errored: 0
Findings: 0 bugs, 0 warnings, 0 info

Findings

No findings -- all journeys completed without issues.

Screenshots

Screenshots are available in the workflow artifacts.

Workflow run: https://github.com/elastic/kibana-exploratory-testing/actions/runs/24749151549

flash1293 · 2026-04-21T17:42:42Z

THanks @viduni94 , definitely better - we'll need to wait for the core team of course for how to proceed here. I'm still puzzled it worked so far... we have lots of tests running serverless that should catch this exact kind of thing, right?

viduni94 · 2026-04-21T18:45:33Z

/ci

viduni94 · 2026-04-21T20:41:15Z

Thanks @viduni94 , definitely better - we'll need to wait for the core team of course for how to proceed here. I'm still puzzled it worked so far... we have lots of tests running serverless that should catch this exact kind of thing, right?

@flash1293 I had to update the implementation slightly to pass the build flavour from the evals plugin as esClient.info() doesn't work as expected when executed in the scope of the user's context/permissions.
Let me know what you think. Tested that this works in the test serverless deployment on this PR.

elasticmachine · 2026-04-21T22:11:18Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 05c6a8c
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-264760-05c6a8c37647

Failed CI Steps

Test Failures

[job] [logs] affected Scout: [ platform / streams_app-stateful-classic ] plugin / local-stateful-classic - Stream data routing - creating routing rules - should show validation errors for invalid stream names verified on the client
[job] [logs] Jest Tests #3 / SelectedFilters should render properly
[job] [logs] Jest Tests #10 / ShareToSpaceFlyout with enableCreateCopyCallout shows a warning callout when the saved object only has one namespace

Metrics [docs]

✅ unchanged

History

💛 Build #431670 was flaky d18d804
💔 Build #431575 failed 66ebdf4

cc @viduni94

TinaHeiligers

config/serverless.oblt.yml still has xpack.evals.enabled: true with a TODO comment to remove before merging. That needs to come out.

The three-tier serverless detection (explicit flag, proactive esClient.info() check, reactive catch-and-retry) is well thought out and the tests cover all three paths plus the caching behavior. The rest looks good to me.

viduni94 · 2026-04-21T23:36:57Z

config/serverless.oblt.yml still has xpack.evals.enabled: true with a TODO comment to remove before merging. That needs to come out.

Thanks for the review @TinaHeiligers
I have already removed xpack.evals.enabled: true in 05c6a8c after testing.

The three-tier serverless detection (explicit flag, proactive esClient.info() check, reactive catch-and-retry) is well thought out and the tests cover all three paths plus the caching behavior. The rest looks good to me.

Thank you 🙏🏻

SrdjanLL

LGTM - code review only!

I think in the long term we should have evals indices registered as system indices in ES (not a blocker at this early stage though).

I've created an issue to track this #264945 - feel free to add more context as needed.

TinaHeiligers

Looks ok to me

…astic#264760) Closes elastic#264845 ## Summary Fixes index template creation on Serverless for indices `kibana-evaluation-datasets`, `kibana-evaluation-dataset-examples`). PR elastic#263096 added `auto_expand_replicas` and `number_of_shards` to index templates in `StorageIndexAdapter`. Serverless ES rejects these settings on non-system indices with an `illegal_argument_exception`, while hidden indices (e.g.: used by Streams) are unaffected because Kibana manages them as system indices. ### Dataset upsert error for Kibana evaluation runs <img width="1247" height="473" alt="image" src="https://github.com/user-attachments/assets/10e75668-7a1d-462e-9594-37fbee0f08e3" /> ### Error in logs: ``` Failed to upsert evaluation dataset: ResponseError: illegal_argument_exception Root causes: illegal_argument_exception: Settings [index.auto_expand_replicas,index.number_of_shards] are not available when running in serverless mode ``` ## Fix The changes were introduced in three tiers to detect serverless environments for index template settings: - Explicit detection - Introduced a new `isServerless` option in `StorageIndexAdapterOptions`. When provided, the adapter skips or includes settings without any extra calls. - Proactive - if `isServerless` is not provided, the adapter calls `esClient.info()` on the first write and checks `version.build_flavor`. The result is cached for the adapter's lifetime. - Reactive - if both above are unavailable (e.g.: `info()` fails due to insufficient privileges), the adapter catches the `illegal_argument_exception` on the first write, retries without settings, and caches the result. The Evals plugin passes `isServerless` explicitly because the evals route handler creates `StorageIndexAdapter` with `esClient.asCurrentUser`, which is scoped to the caller's API key. This API key may lack the monitor cluster privilege needed for `esClient.info()`, making tier 2 unreliable. There `buildFlavor` is passed from the plugin context. ## Test Plan - [x] Deploy the fix to a serverless project from this PR - [x] Create a config file (e.g.: `config.testcluster.json`) and add the serverless project URL as the dataset target - [x] Run evals with `node scripts/evals start --suite significant-events --project eis-anthropic-claude-4-6-sonnet --judge eis-google-gemini-3-1-pro --export-profile local --datasets-profile testcluster` ### With this fix, the dataset upsert works as expected <img width="1531" height="877" alt="image" src="https://github.com/user-attachments/assets/84c2a5cd-138b-457e-85d3-bd87bff4867c" /> <img width="1710" height="556" alt="image" src="https://github.com/user-attachments/assets/bbfeb03a-405f-4551-8326-e12b0192d332" /> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels.

viduni94 added 2 commits April 21, 2026 11:39

[StorageIndexAdapter] Fix index template creation on Serverless

f1d8565

Update test

8773e09

viduni94 requested review from a team as code owners April 21, 2026 15:44

viduni94 added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting ci:project-deploy-observability Create an Observability project labels Apr 21, 2026

viduni94 requested a review from flash1293 April 21, 2026 15:50

Temporarily enable the eval plugin

227852b

viduni94 requested a review from a team as a code owner April 21, 2026 15:54

viduni94 requested a review from rgodfrey-elastic April 21, 2026 15:54

viduni94 commented Apr 21, 2026

View reviewed changes

Add todo

4557df4

viduni94 commented Apr 21, 2026

View reviewed changes

Comment thread config/serverless.oblt.yml Outdated

Update comment

3338bf4

macroscopeapp Bot reviewed Apr 21, 2026

View reviewed changes

Comment thread config/serverless.oblt.yml Outdated

Check for serverless proactively

66ebdf4

viduni94 marked this pull request as draft April 21, 2026 17:58

Pass build flavour from plugin

d18d804

viduni94 self-assigned this Apr 21, 2026

viduni94 marked this pull request as ready for review April 21, 2026 20:41

viduni94 requested review from a team as code owners April 21, 2026 20:41

infra-vault-gh-plugin-prod Bot deleted a comment from elasticmachine Apr 21, 2026

Remove temp config

05c6a8c

viduni94 removed request for a team and rgodfrey-elastic April 21, 2026 20:42

viduni94 mentioned this pull request Apr 21, 2026

[StorageIndexAdapter] Index template creation fails on Serverless for non-system indices #264845

Closed

TinaHeiligers reviewed Apr 21, 2026

View reviewed changes

SrdjanLL mentioned this pull request Apr 22, 2026

[kbn-evals] Register evals indices as system indices in Elasticsearch #264945

Open

SrdjanLL approved these changes Apr 22, 2026

View reviewed changes

TinaHeiligers approved these changes Apr 22, 2026

View reviewed changes

viduni94 merged commit 6097295 into elastic:main Apr 22, 2026
18 checks passed

kibanamachine added the v9.5.0 label Apr 22, 2026

Conversation

viduni94 commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dataset upsert error for Kibana evaluation runs

Error in logs:

Fix

Test Plan

With this fix, the dataset upsert works as expected

Checklist

Uh oh!

github-actions Bot commented Apr 21, 2026

🤖 GitHub comments

Uh oh!

viduni94 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

flash1293 commented Apr 21, 2026

Uh oh!

viduni94 commented Apr 21, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Run Metadata

Findings

Screenshots

Uh oh!

flash1293 commented Apr 21, 2026

Uh oh!

viduni94 commented Apr 21, 2026

Uh oh!

viduni94 commented Apr 21, 2026

Uh oh!

elasticmachine commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

TinaHeiligers left a comment

Choose a reason for hiding this comment

Uh oh!

viduni94 commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SrdjanLL left a comment

Choose a reason for hiding this comment

Uh oh!

TinaHeiligers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

viduni94 commented Apr 21, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Apr 21, 2026 •

edited

Loading

elasticmachine commented Apr 21, 2026 •

edited

Loading

viduni94 commented Apr 21, 2026 •

edited

Loading